Neural Attention Models for Sequence Classification: Analysis and Application to Key Term Extraction and Dialogue Act Detection

摘要

Recurrent neural network architectures combining with attention mechanism, orneural attention model, have shown promising performance recently for the tasksincluding speech recognition, image caption generation, visual questionanswering and machine translation. In this paper, neural attention model isapplied on two sequence classification tasks, dialogue act detection and keyterm extraction. In the sequence labeling tasks, the model input is a sequence,and the output is the label of the input sequence. The major difficulty ofsequence labeling is that when the input sequence is long, it can include manynoisy or irrelevant part. If the information in the whole sequence is treatedequally, the noisy or irrelevant part may degrade the classificationperformance. The attention mechanism is helpful for sequence classificationtask because it is capable of highlighting important part among the entiresequence for the classification task. The experimental results show that withthe attention mechanism, discernible improvements were achieved in the sequencelabeling task considered here. The roles of the attention mechanism in thetasks are further analyzed and visualized in this paper.